Repository navigation
[AMD] fix(docker): drop the obsolete ROCm Megatron fused-kernels patch - #2768
Merged
yushengsu-thu merged 1 commit intoAug 27, 2026
Conversation
radixark/Megatron-LM@miles-main now carries upstream NVIDIA commit 3aa73966
("Guard non-core imports", radixark#3993), which deleted megatron/legacy/fused_kernels
along with the fused_kernels.load(args) call in _compile_dependencies(). The
ROCm patch guarded that loader with a torch.version.cuda check, so its only
target file no longer exists and `git apply` fails:
error: megatron/legacy/fused_kernels/__init__.py: No such file or directory
That breaks Dockerfile.rocm step 17 for every ROCm variant, taking down the
sglang nightly ROCm image releases for rocm700-mi35x, rocm700-mi30x and
rocm720-mi35x. Nothing on miles-main loads the legacy fused kernels anymore, so
the guard is dead code: remove the patch, its COPY and `git apply`, and the
apply/verify block the ROCm PR CI ran for a Megatron-PR override.
michaelzhang-ai
marked this pull request as ready for review
August 26, 2026 21:25
michaelzhang-ai
requested review from
guapisolo,
yueming-yuan and
yushengsu-thu
as code owners
August 26, 2026 21:25
michaelzhang-ai
added a commit
to sgl-project/sglang
that referenced
this pull request
Aug 26, 2026
Throwaway test branch: repoints the miles checkout at the fork branch carrying radixark/miles#2768 and replaces the Docker Hub login/push steps with an image inspect, so the release build is exercised end to end without publishing a tag. Not for main - delete this branch once #2768 merges. Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com>
Contributor
Author
|
@Arist12 @yushengsu-thu @guapisolo @yueming-yuan please help review. Thanks! |
HaiShaw
approved these changes
Aug 26, 2026
Contributor
|
LGTM |
1 similar comment
Contributor
|
LGTM |
yushengsu-thu
approved these changes
Aug 27, 2026
2 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every ROCm image build fails at
Dockerfile.rocmstep 17:docker/amd_patch/latest/megatron.patchadds a ROCm guard (if not torch.version.cuda: return) to Megatron's legacy fused-kernel loader.radixark/Megatron-LM@miles-mainnow carries upstream NVIDIA commit3aa73966("Guard non-core imports", NVIDIA/Megatron-LM#3993), which deleted the wholemegatron/legacy/fused_kernels/package and removed thefused_kernels.load(args)call from_compile_dependencies()inmegatron/training/initialize.py. The patch therefore has no target file andgit applyfails hard.This takes down the nightly ROCm image releases for all three variants —
rocm700-mi35x,rocm700-mi30xandrocm720-mi35x— which are published fromsgl-project/sglang:The first red nightly is Aug 25. The Aug 24 12:00 UTC run still passed, and #2734 ("docker: point MEGATRON_BRANCH back at miles-main") merged at Aug 24 20:01 UTC, right after it —
miles-mainhad been moved onto the newer NVIDIA dev base by then, and the interimmiles-main-20260819branch has since been deleted.Dockerfile.rocmhas noMEGATRON_COMMITarg, so there is no way to pin around this from the calling workflow; it has to be fixed here.Fix
Nothing on
miles-mainloads the legacy fused kernels anymore, so the guard is dead code. This removes:docker/amd_patch/latest/megatron.patch, plus itsCOPYand thegit applyinDockerfile.rocm_run-ci-rocm.ymlthat ran when a Megatron-PR override was selected — with the patch file gone thatgit apply --checkwould fail too, and it no longer has anything to verifytests/ci/test/test_run_suite.py::test_megatron_override_preserves_rocm_patchpinned that block's ordering, so it becomestest_megatron_override_installs_the_checked_out_ref_unpatched, asserting the override installs the checked-out ref with no patch step.Result: all three ROCm release images build green
Both sglang nightly release workflows were dispatched against
9d5e695(this branch's head) and all three variants succeeded:rocm700-mi35xrocm/sgl-dev:miles-rocm700-mi35x-20260826, 67.9 GBrocm700-mi30xrocm/sgl-dev:miles-rocm700-mi30x-20260826, 65.6 GBrocm720-mi35xrocm/sgl-dev:miles-rocm720-mi35x-20260826, 84.2 GBThose durations are in line with the last known-good nightlies (17m for 7.0, 21m for 7.2 on Aug 24), so nothing regressed on build time.
The previously failing stage now reads
[sglang 16/37] RUN rm -rf /root/Megatron-LM && git clone --recursive -b miles-main ... && pip install -e .— one stage fewer, since the patchCOPYis gone — and completes. All 42 buildx stages finish and the image exports (naming to docker.io/rocm/sgl-dev:miles-rocm720-mi35x done).How it was run
The dispatch used a throwaway
sgl-project/sglangbranch carrying the two real nightly workflows with exactly two edits each: the miles checkout repointed at this branch, and the Docker Hub login plus the twodocker pushsteps replaced bydocker image inspect. Runner (amd-docker-scale),prodenvironment, variant matrix,docker/build.pyinvocation and date tagging were all untouched, so this is the release path minus publication. Nothing was pushed torocm/sgl-dev. The harness branch has since been deleted; the run logs above remain.Also checked
megatron/legacyis absent from the currentmiles-maintree (235952df); the only survivingmegatron.legacyreferences are the unrelatedfp16_deprecatedloss-scaler shims incheckpointing.py, so dropping the guard changes no live code path.pytest tests/ci/test/test_run_suite.py— 89 passed._run-ci-rocm.ymlstill parses as YAML; no reference tomegatron.patchsurvives anywhere in the tree.docker/image_inputs.pydoes not hashdocker/amd_patch/, so the rebuild-skip label is unaffected.docker pushsteps, which this change does not touch.Note
docker/amd_patch/sglv0.5.10/megatron.patchis a copy of the same obsolete patch and is referenced by nothing in the repo. Left alone here to keep the change minimal; happy to delete it in this PR if preferred.